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Abstract 

Facilitating student growth and development in diagnosing and solving 
technical problems remains a challenge for technology and engineering 
educators. With funding from the National Science Foundation, this team of 
researchers developed a self-guided, computer-based instructional program to 
experiment with conceptual mapping as a treatment to improve undergraduate 
technology and engineering students’ abilities to develop a plan of action when 
presented with real technical problems developed in collaboration with 
industrial partners. The pilot-testing and experimental portions of the project 
confirmed that the subjects found the training interesting and useful. However, 
the experimental data did not support the conclusion that the treatment caused an 
increase in diagnostic performance. Analysis of individual data demonstrated 
that for a portion of the sample, significant gains occurred. Qualitative analysis 
demonstrated that the majority of the students treated the experience as a trivial 
academic exercise, which seriously limited their efforts. A major step forward 
occurred in the ability to automatically compare student maps with the expert’s 
reference map using a modified version of the similarity flooding algorithm. 
Student feedback was based on this automatic comparison. In all, the experience 
has encouraged the team to continue and expand the use of conceptual mapping 
as a tool for improving diagnostic and problem-solving skills. 

Keywords: diagnostic skills; conceptual mapping; self-guided instructional 
design; similarity flooding 


Introduction 

For some time, technology and engineering educators have focused on 
improving students’ cognitive and metacognitive abilities, especially in the areas 
of design and problem solving. It is clear that these abilities are essential for 
personal and professional success. However, an argument can be made that 
more work is needed. This study was an attempt, supported by funding from the 
National Science Foundation, to develop and pilot test computer-based, self- 
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paced diagnostic skills training using conceptual mapping techniques developed 
by Novak (1977). Cognitive abilities are highly successful predictors of job 
performance, and the development of advanced cognition skills is critical to 
perform any diagnostic assignment. According to Gabbert, Johnson, and 
Johnson (2001), one of the ways to address cognitive development is “by 
examining the strategies students use to complete learning tasks” (p. 267). 
Causal-model theory argues “that, in principle, people are sensitive to causal 
directionality during learning”; our learning does not involve only “acquiring 
associations between cues and outcomes irrespective of their causal role” 
(Waldmann, 2000, p. 53). It is predicted by associative theories that in some 
learning conditions, our representations are fundamentally flawed. There is 

evidence that learning from experience seems more of a process of 
negotiation in which thinking, reflecting, experiencing and action are 
different aspects of the same process. It is negotiation with oneself and in 
collaboration with others that one may actually form the basis of learning. 
(Brockman, 2004, p. 141) 

Interestingly, this applies as well to the informal learning that occurs in such 
technical situations as help desk support for information systems. Haggerty 
(2004) noted that the support and implementation of structured and systematic 
problem-solving processes and detailed verbal modeling of explanations about 
the problem are critical to the development of advanced cognition and effective 
outcomes because they will also increase self-efficacy and satisfaction. 

Diagnostic learning (i.e., analyzing causes based on information about the 
effects) needs to be differentiated from predictive learning (i.e., determining 
effects based on specific causes). “An important feature of diagnostic inference 
is the necessity of taking into account alternative causes of the observed effect” 
(Waldmann, 2000, p. 55). Given that the effects (symptoms) have already 
occurred and are not necessarily actively recognized by the learner, “diagnostic 
learning is a test case for humans’ competency to form and update mental 
representation in the absence of direct stimulation” (Waldmann, 2000, p. 73). 

Some technical problems are well defined with a clear goal: There is a 
definite cause and outcome and a proven algorithm to ensure that the problem is 
solvable. However, many technical problems are ill-defined: The cause or 
causes, what constitutes relevant data, and the steps to be taken are unclear. 
Consequently, there is a need for advanced cognitive skills such as analytical, 
creative, and practical thinking when diagnosing technical problems. Writing 
from an organizational perspective, Okes (2010) states that a “problem is often 
not a single problem but many different problems” (p. 38). In such cases, the 
diagnosis will be difficult since there are likely to be multiple causes. The ability 
to diagnose multiple causes of a common effect versus a common cause for 
multiple] effects requires the technician to learn about these fundamental causal 
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relationships correctly. And this ability is improved through practical experience 
and the ability to evaluate the causes and solutions of problems. Waldmann 
(2000) believes that a combination of technical expertise and logical and 
creative thought processes are essential for diagnosing a problem. Okes (2010) 
makes a distinction between creative problems and analytical problems: With 
creative problems, multiple solutions are necessary; however, analytical 
problems require “the right solutions [that] will not be known until a proper 
diagnosis is done. [For analytical problems,] it is the [utilization of a] diagnostic 
process, known as root cause analysis, which finds the causes” (p. 37). 

Technical Diagnostic Work 

“Most people never receive training in root cause analysis, and those who 
are experts have often learned it through years of experience diagnosing a wide 
range of problem situations” (Okes, 2010, p. 36-37). One of the ways in which 
diagnostics skills are developed is through the understanding of cause and effect 
relationships when performing diagnostics. Typically, technical workers learn 
“cause and effect relationships resembling symptom-cause troubleshooting 
charts which they held in memory for use in subsequent troubleshooting” 

(Green, 2006, Abstract, p. 2). 

Troubleshooting requires the technician to utilize problem-solving skills. 
According to Sharit and Czaja (2000), this is one of the most complex cognitive 
processes. Haggerty (2004) writes that providing “technical support [is] . . . one 
mechanism by which [information system] users can gain the necessary 
knowledge, skills and abilities to use their technology [the system] effectively” 
(p. iii). As Haggerty (2004) states, “effective support is characterized by a 
timely and well structured problem solving process where a knowledgeable, 
sympathetic and patient analyst provides thorough and specific information and 
explanations, matched to the user's demonstrated level of ability, according to 
the needs of the specific technical problem” (p. iii). Technical workers, 
however, often encounter complex, ill-structured problems in their professional 
efforts to solve technical problems and do not have access to any form of 
technical support. Three major issues relative to organizational culture’s impact 
on problem solving are identified by Okes (2010), including: how people view 
problems (e.g., someone to blame), who will be called upon to diagnose and 
solve the problem, and the ratio of the number of problems to the available 
personnel to solve them. 

Instructional Techniques 

Most would agree with Stoyanov and Kirschner (2007) that “solving 
problems is considered an important competence of students in higher 
education” (p. 49) and that higher education institutions must take on this 
important task. What seems to be at issue is how to go about the task effectively. 
Historically, problem-solving instruction has focused mostly on well-defined 
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problems and a rather simple, step-by-step approach. However, most situations 
that are viewed as problems are typically not well defined, nor do people tend to 
be successful in solving them using a simplistic problem solving heuristic. What 
seems to be needed is practice with ill-defined problems within a supportive and 
instructional environment (Stoyanov & Kirschner, 2007). 

To develop diagnostic skills, “the knowledge that one should analyze the 
problem situation, generate ideas, select the most appropriate, and then 
implement and evaluate is necessary, but not sufficient” (Stoyanov & Kirschner, 
2007, p. 50). It is also important to know how to process information, form and 
test hypotheses, and make choices based on the data (Johnson, 1994). “The 
selection and application of these procedures, techniques, and tools depends to a 
large extent on the desired outcomes of problem-solving determined by the 
nature of ill-structured problems and the cognitive structures and processes 
involved in solving them” (Stoyanov & Kirschner, 2007, p. 50). “Instructional 
design should determine the most effective and efficient conditions of providing 
both process and operational support to solving ill-structured problems” (p. 50). 

Writing on the topic of professional development, Mayer (2002) defines 
problem-based training as providing “realistic problems and the solutions of 
these problems in a variety of situations” (p. 263). He lists the four types of 
problem-based approaches enumerated by Lohman (2002) that “can be 
presented in computer-based environments, book-based environments, or live 
environments” (Mayer, 2002, pp. 263-264): (1) case study , (2) goal-based 
scenario , (3) problem-based learning , and (4) action learning (p. 264). He 
theorizes that 

“there are three cognitive steps in problem solving by analogy: (1) 
recognizing that a target problem is like a source problem you already 
learned to solve, (2) abstracting a general solution method, and (3) mapping 
the solution method back onto the target problem” (p. 267). 

More research is needed “to understand how each of the problem-based training 
methods [in Lohman’s (2002) article] supports the processes described [the three 
cognitive steps]” (Mayer, 2002, p. 267). The use of graphical structures to help 
make sense of information is important to problem solving and systems 
thinking. 

Novak, the person credited with the development of conceptual mapping, 
argues “that the central purpose of education is to empower learners to take 
charge of their own meaning making” (Novak, 2010, p. 13). His work in 
conceptual mapping is based on the theoretical work of Ausubel, Piaget, and 
Vygotsky. Initially developed as a means to collect data about children’s 
knowledge, his research quickly led to the conclusion that concept mapping was 
useful in helping students learn and helping teachers to organize instruction. It 
has also been demonstrated to be a useful tool for evaluating what has been 
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learned. Recently, he has applied mapping as planning and problem solving in 
corporate settings. 

Siau and Tan (2006) identify “three popular cognitive mapping 
techniques—causal mapping, semantic mapping, and concept mapping” (p. 96). 
The term cognitive map is developed in psychology as a means of describing an 
individual's internal mental representation of the concepts and relations among 
concepts. Cognitive mapping techniques are used in identifying subjective 
beliefs and to represent these beliefs externally. “The general approach is to 
extract subjective statements from individuals, within specific problem domains, 
about meaningful concepts and relations among these concepts and then to 
describe these concepts and relations in some kind of diagrammatical layout 
[Swan, 1997]” (Siau & Tan, 2008, p. 100). This internal mental representation is 
used to understand the environment and make decisions accordingly. “Causal 
mapping [emphasis added] is the most commonly used cognitive mapping 
technique by researchers when investigating the cognition of decision-makers in 
organizations [Swan, 1997]” (Siau & Tran, 2006, p. 100) because it allows an 
individual to interpret the environment with salient constructs. The theory 
argues that individuals, with their own personal system of constructs, use it to 
understand and interpret events. “Semantic mapping [emphasis added], also 
known as idea mapping, is used to explore an idea without the constraints of a 
superimposed structure [Buzan, 1993]” (Siau & Tran, 2006, p. 101). With 
semantic maps, an individual will begin at the center of the paper with the 
principal idea and work outwards in all directions. This produces an expanding 
and organized structure consisting of key words and key images. Concept 
mapping is another cognitive mapping technique, which “is a graphical 
representation where nodes represent concepts, and links represent the 
relationships between concepts” (Siau & Tan, 2006, p. 101). Concept mapping 
is an integral part of systems thinking. 

Systems Thinking 

Batra, Kaushik, & Kalia (2010) define systems thinking as 

a holistic way of thinking, fundamentally different from that of traditional 
forms of analysis in which the observer considers himself the part of reality 
as a whole system. System[s] thinking resists the breaking down of 
problems into its component parts for detailed examination and focuses on 
how the thing being studied interacts with the other constituents of the 
system. . . . This means that instead of taking smaller and smaller parts or 
view[s] of the system taken for study, it actually works by expanding its 
view by taking into account larger and larger numbers of views or parts of 
the system, (p. 6) 

“Systems thinking is increasingly being regarded as a ‘new way of thinking’ 
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to understand and manage the ‘natural’ and ‘human’ systems associated with 
complex problems . . . (Bosch et al., 2007a)” (Nguyen, Bosch, & Maani, 2011). 
When employing systems thinking to deal with real problems, it is necessary for 
the trainer to classify real-world problems by system language because the 
classification can help students or trainees find an appropriate method and 
methodology to deal with specific problematic situations. According to Batra et 
al. (2010), 

The character / nature / approach of systems thinking makes it extremely 
effective on the most difficult types of problems to solve. . . . Some of the 
examples in which system thinking has proven its value include: complex 
problems that involve helping many actors [learners] see the “big picture” 
and not just their part; recurring problems or those that have been made 
worse by past attempts to fix them; issues where an action affects (or is 
affected by) the environment surrounding the issue, either the natural 
environment or the competitive environment; and problems whose solutions 
are not obvious, (p. 6) 

Batra et al. (2010) discovered that systems thinking leads to: “in-depth 
search of problem contributors by finding out further reasons for those problems 
and lead to actual reasons of the same,” “solutions] to all kinds of problems by 
considering them as a whole system” (p. 10), finding the root-cause of a 
problem by exploring not just analyzing the problem and making assumptions, 
and “permanent solutions of problems by acting on all possible reasons 
simultaneously [instead of just making] plans to solve the problem by removing 
the reasons one by one as per the plan formation” (p. 11). 

Evaluation of Diagnostic Skills 

In order to acquire skills, learning has to take place. According to 
Kontogiannis and Maoustakis (2002), “most research in artificial intelligence 
and machine learning has . . . underplayed issues of problem formulation, data 
collection and inspection of the derived knowledge structures (Langley and 
Simon 1995)” (p. 117). They further state that the stages of inspection and 
evaluation of knowledge structures are significant because “by making 
knowledge structures easier to understand or comprehend, we are in a better 
position to meet criteria of validation, generalization and discovery.” (p. 133). 
Basically, the issue of comprehensibility must be part of the evaluation process 
for technical applications. The learner has to be able to input and output 
knowledge. Comprehensibility should address transferability to a new context in 
the future. With the increased use of computer-based diagnostic skills, 
evaluation is not done separately from the ability of humans to interact with the 
computer to perform problem-solving tasks. 
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Kontogiannis’ and Maoustakis’ (2002) “informal approach for refining and 
elaborating knowledge structures” (p. 120, Figure 1) provides some insight on 
what to consider when performing evaluations of diagnostic skills. The model 
illustrates the significance of the individual to become an expert in the field to 
effectively perform diagnostics. It is imperative that the expert is able to employ 
several diagnostic strategies. This entails “fault-finding strategies and 
knowledge structures [, which] are dependent upon the amount and kind of data 
that are available—for example, data regarding equipment reliability data, 
direction and rate of change of process variables, sequences of change, etc.” (p. 
120). Their model demonstrates the significance of identifying any weakness in 
the process at the analytical stage. The role of the expert is to determine if the 
process for solving the problem was justifiable in terms of the principles of 
operation. At the modification stage, the ability of the learner “to impose a 
hierarchical structure upon the” (p. 123) process is crucial to problem solving. 
This would entail “providing descriptions of groups of faults that relate 
functionally to each other” (p. 123). Moreover, the creation of subordinate goals 
and plans are important as they are “high-level objectives or concepts .... By 
progressively breaking up all goals into plans or sequences of checks, the overall 
task becomes easier to achieve” (Kontogiannis & Maoustakis, 2002, p. 123). 
Kontogiannis and Maoustakis further explain that “the distance or gap between 
the top goal and the available responses [, verbal statements of the problem 
made at a high or detailed level,] becomes smaller” (p. 123). The subjects of 
their study were required to participate in fault-finding activities. This 
demonstrated the level to which they were able to transfer what was learned. In 
order to study subjects’ diagnostic skills, the authors created three modules: 
“Module 1—Training,” “Module 2—Fault finding test in the manual mode” (p. 
128), and “Module 3—Transfer to diagnosis in the automatics mode” (p. 129). 
Their findings indicated “that deep goal structures facilitate both the acquisition 
and the transfer of knowledge” (p. 132). 

Problem and Purpose of the Study 

Research to date supports the assertion that domain-specific problem¬ 
solving skills can be developed by educational and training programs. The more 
experience an individual has with a system and with solving problems that occur 
in the system, the more proficient a problem solver that individual becomes. 
What is less certain is the extent to which general problem-solving instruction 
can result in improvements in near and far transfer of knowledge to novel 
problem situations. There is strong evidence that the application of conceptual 
mapping improves learning, memory, and application of knowledge. It is also 
clear that conceptual mapping enhances systemic understandings. 

This project was designed to test the premise that the use of conceptual 
mapping techniques to force in-depth analysis of technical problems and the 
creation of a process map of the student’s intentions during the diagnosis phase 
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of problem solving would increase the student’s level of accuracy when 
compared to the process used by experts in the particular system. It was argued 
that developing this expertise in mapping would help the students become more 
agile in their thinking regarding ill-defined problems. 

A quasi-experimental model was used to test the null hypothesis Ho: (Iai = 
PA 2 = ftA 3 = J-IA4• In other words, there would be no statistically significant 
difference between the four groups in terms of the mean difference in percentage 
accuracy for the two trials. 


Method 

The project staff developed a 2-hour, computer-based training program 
using the Lectora authoring software. The first hour of this training was 
designed to introduce students to systems and troubleshooting problems. During 
this phase of the training, the subjects were taught to use the Visual 
Understanding Environment (VUE) open-source software to develop conceptual 
and process maps. Also during this phase, the subjects were given a “simple” 
technical problem and asked to map out a plan of action. Using a similarity 
flooding algorithm, the students’ maps was then electronically compared to an 
expert’s map that was already encoded. The information developed by the 
algorithm was returned to the subjects as feedback so that they would know how 
well they did. 

The second phase of the training took the form of two technical problems 
developed in collaboration with industrial partners. The first problem involved 
an electrical grid distribution system, and the second problem focused on a 
malfunctioning heat exchanger system. In both cases, the subjects were provided 
with a description of the system and tables containing operational information. 
From the information presented, the subjects were asked to analyze the system 
and develop a process map for addressing the apparent malfunction. 

The evaluation of this project included pilot-testing the training program 
after the completion of the first hour, after the completion of the electrical grid 
problem, and after completion of the full training program. The subjects were 
provided the program and all necessary software on a memory stick and 
typically completed the training on a laptop. 
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Figure 1. Sample expert process map (practice problem). 
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Instrumentation 

A feedback questionnaire was developed to provide subjects the opportunity 
to express their reactions to the training. The Likert-type questions were similar 
to end-of-course student evaluations of instruction. In this case, the focus was on 
the “look and feel” of the program and the extent to which the students 
perceived the training to be interesting, relevant, and helpful. Subjects were also 
encouraged to provide open-ended comments. 

Population and Sample 

The population for this study was junior and senior undergraduates enrolled 
in engineering, engineering technology, and closely related majors. The sample 
was drawn from students enrolled in the College of Technology at Indiana State 
University (the Principle Investigators’ home institution) and four other 
universities. Faculty members involved with Indiana State University’s PhD in 
Technology Management Consortium and faculty members at the four other 
institutions accepted invitations to participate. The sample consisted of the 
classes of those faculty members. 

There were four treatments, and the groups were randomized and assigned 
to the groups of students in the order in which the faculty of these groups agreed 
to participate. Typically, faculty members would volunteer a particular class that 
they taught because the training was perceived to be relevant to the focus of the 
particular course. Subject participation was voluntary, confidential, and 
anonymous. 

Data Collection 

Experimental data was collected automatically as the subjects completed the 
training. The primary dependent variable for the experiment was the difference 
in percent accuracy for the two attempts for each problem. Upon completion and 
submission of the subject’s map, the map was compared to the expert’s map, 
and percentage accuracy was reported. The subject received this percentage as 
feedback. One group only received its scores (control condition). The other three 
groups were given a secondary form of additional feedback: metacognitive cues, 
review of the expert map, and both. All subjects then were given the opportunity 
to revise and resubmit their maps. The resubmission was also evaluated for 
accuracy, and the percentage was reported to the students and recorded. 

After completing the training program, the subjects were given a 
“satisfaction” questionnaire to provide feedback regarding their reactions to the 
training. There was a range of 1-4 for each of the eight questions asked. 
Consequently, the overall range for the satisfaction score was 8 to 32 points. 

Data Analysis 

Student satisfaction data was analyzed using only descriptive statistics that 
resulted in a mean and standard deviation for each question and the total for all 
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eight questions (i.e., overall satisfaction). Experimental data analysis included 
testing the differences in percentage accuracy between the first draft and second 
draft of the subject’s process maps. In addition, one-way analysis of variance 
was used to determine if there were any differences in mean percentage 
accuracy based on treatment type, university where the subject was studying, 
and major. 


Results 

Sample 

A total of 130 subjects participated in the experimental portion of this 
project. Table 1 provides a summary of the demographic data collected. 


Table 1 

Demographic Data 



Gender 

Major 


University 

Female 9 

(7%) 

Mech. Eng. Tech. 

58 

(44.6%) 

ISU 

72 

(55.4%) 

Male 

106 

(81.5%) 

Const. Mgmt. 

35 

(26.9) 

BGSU 

35 

(26.9%) 

NR 

15 

(11.5%) 

Info. Tech. 

15 

(11.5%) 

IUPUI 

11 

(8.5%) 



Mech. Eng. 

11 

(8.5%) 

PU 

8 

(6.1%) 



Pack. Tech. 

10 

(7.7%) 

RHIT 

4 

(3.1) 



Elec. & Comp. 

Tech 

1 

(.8%) 




The home institution for this project was Indiana State University. As a result, 
the majority (55.4%) of the subjects were from that university. Subjects enrolled 
in mechanical engineering (8.5%) and mechanical engineering technology 
(44.6%) comprised 53.1% of the total. The majority (81.5%) of the subjects 
were male. 
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Satisfaction 

Overall, on a scale of 8-32 points, the mean score for subject satisfaction 
with the training was 25.62 (S n -i = 3.25). Each subject’s ratings were averaged 
across the eight questions. The overall mean for all subjects was 3.2. Table 2 
provides a summary for each of the eight questions. 

Table 2 

Mean and Standard Deviation for the Eight Satisfaction Questions 


Item M SD 


1. The computer based training was interesting 

3.09 

0.67 

2. The screen design was reasonably attractive 

3.15 

0.55 

3. The screen layout was logical (i.e., made 
sense) 

3.27 

0.57 

4. All of the program buttons worked as 
expected 

3.21 

0.70 

5. It was easy to navigate my way through the 
program 

3.29 

0.66 

6. The content of the training program was 
meaningful 

3.12 

0.64 

7. This training would be useful to anyone in a 
technical career 

3.27 

0.67 

8. Overall, this was a high quality, 
professional experience 

3.22 

0.63 


Of the 126 subjects to complete the satisfaction inventory, only 28 (22%) had an 
overall mean satisfaction rating below 3.0. The mean satisfaction scores ranged 
from a low of 1.75 (one subject) to a high of 4.0 (four subject). 

The majority of the subjects chose not to provide any written comments. Of 
the 26 subjects who did comment, 11 (42%) made positive comments, and 15 
(58%) made negative comments. The negative comments focused on parts of the 
program that did not work as expected and on needing more time to complete 
the training. One participant thought that the training would be stronger if it 
were more challenging. 

Experimental Validation 

Subjects were offered the opportunity to provide a process map for three 
problems during the training. The first problem was used as an orientation to the 
process that they would be using for the two main problems. For the first 
problem, the subjects were only given the opportunity to provide one map. 

Based on a comparison with the expert’s map for that problem, the overall mean 
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was 18% accuracy (S n -i = .16). In other words, the subjects found the task 
somewhat challenging. 

Taken as a whole, the subjects who completed the second problem (i.e., 
electrical power grid problem) scored an average of 18.8% (S n -i = .042) on their 
first submission and 15.4% (S n -i = .043) on their second attempt. A paired 
samples f-test revealed that the difference was statistically significant (f<i 29 ) = 
2.919, p < .01), although the reverse of expectations. Subjects who completed 
the third problem (i.e., heat exchanger problem) scored on average 11.7% (S„-i = 
.021) for their first submission and 12.6% (S n -i = .03) for their second 
submission. This was not found to be a statistically significant difference (f(i 28 j = 
-1.17, p > .05). In all cases, percent accuracy was based on 100% (i.e., perfect 
match). 


Table 3 

Mean Differences for the Control Group and Three Treatments for Both 
Problems 


Group 

n 

Diff_l 

Diff_2 

M 

SD 

M 

SD 

1 

35 

-.06214 

176141 

-.00243 

.014368 

2 

32 

-.00813 

123798 

.04019 

.129542 

3 

35 

-.02497 

088989 

-.01091 

.101700 

4 

28 

-.03775 

117151 

.00764 

.026403 

Total 

130 

-.03358 

.131248 

.00795 

.085689 


The null hypothesis was that the mean difference for the three treatments 
would be equal to one another and the control group. Based on a one-way 
analysis of variance, the null hypothesis could not be rejected for the electrical 
grid problem (Fa, 126 ) = 1.014, p = .389) nor for the heat exchanger problem {Fa, 
126 ) = 2.315, p = .079). As can be seen, the mean difference for the two 
submissions for the heat exchanger problem was most pronounced. In addition, 
no statistically significant differences were found based on gender, major, or the 
university at which the students were studying. 

Although group data confirmed that by and large the subjects did not 
perform well on the three problems included in this training, analysis of the top 
performing subjects demonstrated that some students did in fact understand the 
problem and the process mapping technique reasonably well, as shown in Table 
4. 
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Table 4 

Individual Subject Performance for Each Problem 


Electric Grid Problem Heat Exchanger Problem 


First Attempt 

Second Attempt 

First Attempt 

Second Attempt 

0.560 

0.734 

0.651 

0.535 

0.461 

0.526 

0.497 

0.692 

0.423 

0.652 

0.491 

0.554 

0.515 

0.751 

0.419 

0.537 

0.493 

0.741 

0.475 

0.648 



0.342 

0.829 


Discussion 

As stated earlier, the purpose of this project was to experiment with the 
development of a self-paced, computer-based program (using the Lectora 
authoring software) to introduce engineering and technology undergraduates to 
conceptual mapping techniques to help enhance their diagnostic skills for 
technical problems. The first hour of this training was designed to introduce 
students to systems and diagnosing problems. This phase covered the use of the 
Visual Understanding Environment software to develop conceptual and process 
maps. Also during this phase, the subjects were given a “simple” technical 
problem and asked to map a plan of action. Using a similarity flooding 
algorithm, the students’ maps were then electronically compared to an expert’s 
map that was already encoded. The information developed by the algorithm was 
returned to the subjects as feedback. The second phase of the program included 
two technical problems developed in collaboration with industrial partners. The 
first problem involved an electrical grid distribution system, and the second 
problem focused on a malfunctioning heat exchanger system. In both cases, the 
subjects were provided with a description of the system and tables containing 
operational information. From the information presented, the subjects were 
asked to analyze the system and develop a process map for addressing the 
apparent malfunction. 

The computer program was pilot-tested after completion of the first hour, 
after completion of the electrical grid problem, and after completion of the full 
training program. During the quasi-experimental phase, the subjects were asked 
to complete a satisfaction survey of the finished product and the overall 
experience. The quasi-experiment involved subjects from five universities and 
six majors in engineering or technology. The subjects were provided the 
program and all necessary software on a memory stick and typically completed 
the training on a laptop. 
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In short, the subjects during pilot testing and during experimentation found 
the training to be interesting (X = 3.09 on a 4-point scale) and useful (X = 3.22 
on a 4-point scale). However, on average, the subjects’ level of performance was 
much lower than expected. It was clear that about 20% of the subjects clearly 
understood and found the mapping technique and the feedback provided to help 
them improve their diagnostic skills. In general, this was not the case. That is, 
performance on the two main tasks was low and remained low, regardless of the 
type of feedback given. 

There were several indications that provide some explanation (i.e., 
limitations of the work to date). Qualitatively, all of the staff noted that during 
testing, the majority of the subjects (a) did not have enough time to complete 
both of the major problem-solving tasks and (b) were treating the exercise as 
any other academic exercise. A good number of the subjects did nothing more 
than complete a single map for the problem and simply resubmitted it after 
reading the feedback. 

The data, feedback from the subjects, and researcher observations revealed 
that the electric grid problem was difficult for students to process. In other 
words, the heat exchanger problem was much more compatible with the 
simulation capabilities of the training program. Students tended to get lost in the 
screens provided as a static simulation of the electric grid. 

Finally, it was clear that at least 2 hours were needed to complete the 
training as designed. In most cases, the exercise was fitted into a regular class 
period (75-90 minutes), typically without the professor in attendance. For most, 
it was not enough time to complete the training and the two problems. 

A major contribution of this work was the development of automatic 
student feedback by programming the software using a similarity flooding 
algorithm (Melnik, Garcia-Molina, & Rahm, 2002). This algorithm provides a 
means of comparing two graphic items such as conceptual maps. The project 
team modified the original algorithm to include an automatic thesaurus for 
synonyms, the use of relative similarity versus full similarity, weighting the 
nodes, and setting threshold values. Each modification significantly improved 
the algorithm’s accuracy, so that it was possible to compare various expert maps 
with themselves and get a 100% matching indication. The team validated the 
algorithm using multiple tests (see Shahhosseini, Ye, Maughan, & Foster, 2014, 
for a complete report of this work). 

Technology and engineering educators at all levels are seeking additional 
ways to help students improve their cognitive and metacognitive reasoning 
skills. This project was an attempt to determine the extent to which a self-paced, 
computer-based program would help novice troubleshooters (i.e., students) learn 
important principles about mapping and diagnostics. The students found the 
training interesting and useful, but additional work is needed to ensure that the 
instruction is taken seriously by students. Further, the researchers will work to 
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expand the work to all stages in the process from problem identification to 
solution and follow-up assessment. 
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